Meta-Learning with Backpropagation

This paper introduces gradient descent methods applied to meta-leaming (leaming how to leam) in Neural Networks. Meta-leaning has been of interest in the machine leaming field for decades because of its appealing applications to intelligent agents, non-stationary time series, autonomous robots, and improved leaming algorithms. Many previous neural network-based approaches toward meta-leaming have been based on evolutionary methods. We show how to use gradient descent for meta-leaming in recurrent neural networks. Based on previous work on FixedWeight Leaming Neural Networks, we hypothesize that any recurrent network topology and its corresponding leaming algorithm(s) is a potential meta-leaming system. We tested several recurrent neural network topologies and their corresponding forms of Backpropagation for their ability to meta-leam. One of our systems, based on the Long Short-Term Memory neural network developed a leaming algorithm that could leam any two-dimensional quadratic function (from a set of such functions} after only 30 training examples.